Skip to content

feat(chapters): trim clips at the end of a source that decodes short - #83

Closed
ziv-lazarov-nagish wants to merge 1 commit into
mainfrom
sign-1242-write_chapters-optionally-trim-clips-at-the-end-of-a-source
Closed

ziv-lazarov-nagish wants to merge 1 commit into
mainfrom
sign-1242-write_chapters-optionally-trim-clips-at-the-end-of-a-source

Conversation

@ziv-lazarov-nagish

@ziv-lazarov-nagish ziv-lazarov-nagish commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Some headers count frames that never decode. A ChapterClip reaching past the last decoded frame makes write_chapters fail with zip() argument 2 is shorter than argument 1. In signer-detection's export this is 9,938 videos of the 2026-10-25 failure report. Today the only way around it is to count_frames first, which decodes the whole source a second time: 21 s on an 8-minute 1080p video, although write_chapters decodes every frame it needs anyway.

Change

write_chapters(..., trim_to_source=True):

  • A clip that ends past the last decoded frame ends there; a clip that starts past it is dropped. If every clip starts past it, it raises.
  • Matroska writes its chapters in the header, before any frame, so a trimmed file is remuxed once (packets copied with their timestamps, no re-encode) with the chapter table and payloads of the clips written. This runs only when something was trimmed.
  • write_chapters returns the clips written (with trim_to_source off, the clips given).

The default is unchanged: a clip past the source's end still raises.

Relation to #82

  • Chapter times and frame timestamps stay on the source timeline; the remux keeps every packet's pts.
  • Each chapter's payload describes the span it actually holds, so Timeline and read_chapters read trimmed files like any other.
  • No filler frames are added.
  • Clips are still checked for order and overlap before anything is written; trimming only shortens or drops clips.

Notes

  • The remux skips the demuxer's empty flush packet by packet.size, as _count_video_packets does. A first version checked dts is None, but when chapter sizes change and change back (e.g. 48 → 16 → 48 px), Matroska demuxes every packet without a dts, and that version wrote an empty file. The new test uses that layout and fails with the dts check.
  • write_chapters grew past ruff's complexity limit, so the clip loop moved into _write_clips and the chapter table into _chapters.

Checks

  • ruff check . passes.
  • pytest: 361 passed. The 4 failures in test_regression.py also fail on main on this Mac: the local FFmpeg rejects -vsync.
  • In signer-detection, an 8-minute 1080p export without the extra count_frames went from 44.6 s to 23.5 s (warm, MPS), with frame-identical output.

Linear: SIGN-1242

🤖 Generated with Claude Code


Note

Medium Risk
Touches core chapter mux/encode paths and changes write_chapters to return a value; remux only runs when trimming occurs but incorrect chapter metadata would break downstream timeline reads.

Overview
Adds trim_to_source to write_chapters so chapter exports can finish when clip ranges exceed what the decoder actually yields (e.g. header frame counts vs. real stream length), without a separate full-source frame count pass.

With trim_to_source=True, clips that run past the last decoded frame are shortened, clips that start after that point are skipped, and the function returns the ChapterClip list that was written (with updated end_frame values). Default behavior is unchanged: out-of-range clips still raise.

Because Matroska chapters are written in the header before muxed video, any trim triggers a packet-copy remux (_remux_with_chapters) so chapter times and JSON payloads match the encoded spans; remux skips zero-size flush packets so variable chapter dimensions still copy correctly.

Clip encoding is refactored into _write_clips / _chapters; _write_clip gains a non-strict mode and returns the number of frames written.

Reviewed by Cursor Bugbot for commit 24ce44a. Bugbot is set up for automated code reviews on this repo. Configure here.

Headers can count frames that never decode. A clip past the last decoded
frame made write_chapters fail with "zip() argument 2 is shorter than
argument 1", so callers had to decode the whole source a second time to
count its frames first.

With trim_to_source=True a clip ends at the last frame that decodes, and
a clip starting past it is dropped. Matroska writes chapters in the
header, so a trimmed file is remuxed once (packets copied with their
timestamps) with the chapter table and payloads of the clips written.
The default still raises. write_chapters now returns the clips written.

SIGN-1242
@AmitMY

AmitMY commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Opened a competing implementation in #87. It keeps the same public behavior and the same overall approach: detect EOF during encoding, then packet-copy remux only when the chapter table needs correcting.

The main differences are:

  • One place owns the EOF policy. _write_clip encodes available frames and returns their count, including zero. _write_clips decides whether to raise, shorten, or drop a clip. This removes the encoder's strict flag and moves the per-clip peek out of the caller. It may be easier to follow because the encoding helper does not need to know the caller's trimming policy.
  • Remuxing cannot clobber a pre-existing <output>.part. This version uses a fixed staging path that PyAV opens for writing and cleanup removes. I reproduced an existing out.mkv.part disappearing after a trimmed export. feat(chapters): trim short sources with centralized EOF handling [SIGN-1242] #87 stages in a unique temporary directory beside the output, so replacement stays on the same filesystem and unrelated files survive. It also closes the original output handle before replacement.
  • More regression coverage and documentation. feat(chapters): trim short sources with centralized EOF handling [SIGN-1242] #87 retains the changing-dimensions regression and adds fractional FPS, explicit chapter boundaries, decoded-frame equality against an explicitly clamped export, EOF in a gap, skipping remux for complete clips, existing-file preservation, and cleanup after remux failure. The README documents the option and return value.

The potential improvement is clearer ownership of the EOF behavior and safer staging, rather than a different algorithm or a claimed speedup. The remux itself still seems justified. Local validation: lint and diff checks pass; the full suite passed with 392 tests, and all 116 chapter tests passed after the final assertion additions.

@AmitMY AmitMY closed this in #87 Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants